Characteristics of hand and machine-assigned scores

Assessment of learning in higher education is a critical concern to policy makers, educators, parents, and students. And, doing so appropriately is likely to require including constructed response tests in the assessment system. We examined whether scoring costs and other concerns with using open-end measures on a large scale (e.g., turnaround time and inter-reader consistency) could be addressed by machine grading the answers. Analyses with 1359 students from 14 colleges found that two human readers agreed highly with each other in the scores they assigned to the answers to three types of open-ended questions. These reader assigned scores also agreed highly with those assigned by a computer. The correlations of the machine-assigned scores with SAT scores, college grades, and other measures were comparable to the correlations of these variables with the hand-assigned scores. Machine scoring did not widen differences in mean scores between racial/ethnic or gender groups. Our findings demonstrated that machine scoring can facilitate the use of open-ended questions in large-scale testing programs by providing a fast, accurate, and economical way to grade responses. Until the turn of the 21st century, most large-scale K-12 and college testing programs relied almost exclusively on multiple-choice tests. There are several understandable reasons for this. It takes much longer to score the answers to essay and other “constructed response” (which are also referred to below as “open-ended” or “free-response”) questions than it does to have a machine scan multiple-choice answer sheets. Thus, hand scoring of essay answers tends to increase the time required to report results. There also are concerns about subjectivity in grading because human readers do not always agree with each other (or even with themselves over time) in the score they assign to an answer. Scoring costs and logistical problems (such as arranging for readers) are much greater with open-ended tests than they are with multiple-choice exams. In addition, score reliability per hour of testing time is generally greater with multiple-choice tests than it is with open-ended ones (Wainer and Thisssen [19] and Klein and Bolus [5]). Nevertheless, there are important skills that can only be assessed (or assessed well) with open-ended measures. This is especially so in higher education. Consequently, college and graduate school Dr. Laura Hamilton from RAND, Professor Richard Shavelson from Stanford University, and Professor George Ku from Indiana University provided many helpful suggestions on earlier drafts of this chapter. Dr. Roger Bolus, a consultant to the Council for Aid to Education, ran the statistical analyses presented in this chapter. GANSK & Associates, 120 Ocean Park Blvd., #609, Santa Monica, CA 90405, USA, e-mail: